Method of operating observation device, observation device, and recording medium

ABSTRACT

An imaging condition set in a first region of a three-dimensional model of a subject and an imaging condition set in a second region of the three-dimensional model are different from each other. A processor of an observation device determines whether or not the imaging condition that has been set in the first region or the second region including a position on the three-dimensional model is satisfied. The position is identified on the basis of a position of an imaging device and a posture of the imaging device. The processor displays observation information on a display on the basis of a result of determination. The observation information represents whether or not the first region or the second region including the position on the three-dimensional model has been observed.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method of operating an observationdevice, an observation device, and a recording medium.

Priority is claimed on Japanese Patent Application No. 2019-054976,filed on Mar. 22, 2019, the content of which is incorporated herein byreference.

Description of Related Art

Industrial endoscope devices have been used for observation andinspection of internal damage, corrosion, and the like of boilers,turbines, engines, pipes, and the like. There is a case in which a largespace such as an aircraft engine or a structure such as a pipe of longdistance are subjected to inspection using an industrial endoscopedevice. In such inspection, it is difficult for a user to correctlyunderstand where observation has been performed and where observationhas not been performed. In many cases, a user performs inspection byusing a more reliable method in order to avoid omitting observation. Forthis reason, there is a situation in which the same inspection region isrepeatedly inspected. In this way, the efficiency of inspectiondecreases.

In PCT International Publication No. WO2016/076262, a method ofresolving this problem is disclosed. A computer of an endoscope devicedetermines whether or not an inspection object (subject) has beenobserved and notifies a user of a result of the determination. Thecomputer determines whether or not the inspection object has beenobserved on the basis of an object distance of the inspection objectseen in a visual field of an endoscope. If this function is provided, auser can understand where observation has been performed and whereobservation has not been performed, and the efficiency of inspectionincreases.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, a method ofoperating an observation device includes an image display step, anestimation step, an identification step, a determination step, and anobservation information display step. The observation device includes aninsertion unit, a storage medium, and a processor. The insertion unitincludes an imaging device configured to acquire an image of a subjectinside an observation object and is configured to be inserted into theobservation object. The storage medium is configured to store an imagingcondition that has been set in each of a first region and a secondregion. The first region is part of a three-dimensional model of thesubject. The second region is other than the first region in thethree-dimensional model. The imaging condition of the first region andthe imaging condition of the second region are different from eachother. The processor displays the image on a display in the imagedisplay step. The processor estimates, in the estimation step, aposition of the imaging device and a posture of the imaging device. Theposition and the posture are a position and a posture, respectively,when the imaging device acquires the image. The processor identifies, inthe identification step, a position on the three-dimensional modelcorresponding to a pixel of the image on the basis of the position andthe posture that are estimated in the estimation step. The processordetermines, in the determination step, whether or not the imagingcondition that has been set in the first region or the second regionincluding the position on the three-dimensional model is satisfied. Theprocessor displays, in the observation information display step,observation information on the display on the basis of a result of thedetermination step. The observation information represents whether ornot the first region or the second region including the position on thethree-dimensional model has been observed.

According to a second aspect of the present invention, in the firstaspect, the method may further include an imaging condition setting stepin which the processor sets the imaging condition to each of the firstregion and the second region.

According to a third aspect of the present invention, in the first orsecond aspect, the method may further include a region setting step inwhich the processor sets the first region and the second region to thethree-dimensional model.

According to a fourth aspect of the present invention, in the thirdaspect, the method may further include a feature point detection step,an association step, a calculation step, and a generation step. Theprocessor detects a feature point in each of a plurality of imagesacquired by the imaging device in the feature point detection step. Theprocessor associates the feature point between the images included inthe plurality of images in the association step. The processorcalculates a position of the imaging device and a posture of the imagingdevice on the basis of the feature point in the calculation step. Theprocessor generates, in the generation step, the three-dimensional modelon the basis of the position and the posture calculated in thecalculation step. Before the region setting step is executed, thefeature point detection step, the association step, the calculationstep, and the generation step may be executed.

According to a fifth aspect of the present invention, in the secondaspect, input of the imaging condition from a user may not be acceptedin the imaging condition setting step.

According to a sixth aspect of the present invention, in the third orfourth aspect, input of information that represents the first region andthe second region from a user may not be accepted in the region settingstep.

According to a seventh aspect of the present invention, in any one ofthe first to sixth aspects, the processor may estimate the position andthe posture on the basis of at least the image and the three-dimensionalmodel in the estimation step.

According to an eighth aspect of the present invention, in any one ofthe first to sixth aspects, the processor may estimate the position andthe posture on the basis of only the image in the estimation step.

According to a ninth aspect of the present invention, in any one of thefirst to eighth aspects, the storage medium may be configured to store anon-observed position that is the position on the three-dimensionalmodel for which it is determined that the imaging condition is notsatisfied. The method may further include a first notificationinformation display step in which, when the non-observed position isbehind a visual field of the imaging device, the processor displaysfirst notification information that represents that there is the firstregion or the second region including the non-observed position on thedisplay.

According to a tenth aspect of the present invention, in any one of thefirst to ninth aspects, the imaging condition may include a plurality ofconditions. The method may further include a condition informationdisplay step in which, when the processor determines that at least onecondition included in the plurality of conditions is not satisfied, theprocessor displays condition information that represents the at leastone condition on the display.

According to an eleventh aspect of the present invention, in any one ofthe first to tenth aspects, the method may further include a recordingstep in which, when the processor determines that the imaging conditionthat has been set to the first region is satisfied, the processorrecords the image in which a region corresponding to the first region isseen.

According to a twelfth aspect of the present invention, in any one ofthe first to eleventh aspects, when the processor determines that theimaging condition is satisfied, the processor may display, in theobservation information display step, the observation informationrepresenting that the first region or the second region including theposition on the three-dimensional model has been observed on thedisplay.

According to a thirteenth aspect of the present invention, in any one ofthe first to twelfth aspects, when the processor determines that theimaging condition is not satisfied, the processor may display, in theobservation information display step, the observation informationrepresenting that the first region or the second region including theposition on the three-dimensional model has not been observed on thedisplay.

According to a fourteenth aspect of the present invention, in any one ofthe first to eleventh aspects, the processor may calculate, in thedetermination step, an evaluation value that represents a degree towhich the imaging condition that has been set in the first region or thesecond region is satisfied. The processor may display the evaluationvalue as the observation information on the display in the observationinformation display step.

According to a fifteenth aspect of the present invention, in thefourteenth aspect, the processor may determine whether or not theimaging condition is satisfied by comparing the evaluation value with athreshold value in the determination step. The method may furtherinclude a second notification information display step in which, whenthe processor determines that the imaging condition is not satisfied,the processor displays second notification information that representsthe first region or the second region including the position on thethree-dimensional model has not been observed on the display.

According to a sixteenth aspect of the present invention, an observationdevice includes an insertion unit, a storage medium, and a processor.The insertion unit includes an imaging device configured to acquire animage of a subject inside an observation object and is configured to beinserted into the observation object. The storage medium is configuredto store an imaging condition that has been set in each of a firstregion and a second region. The first region is part of athree-dimensional model of the subject. The second region is other thanthe first region in the three-dimensional model. The imaging conditionof the first region and the imaging condition of the second region aredifferent from each other. The processor is configured to display theimage on a display. The processor is configured to estimate a positionof the imaging device and a posture of the imaging device. The positionand the posture are a position and a posture, respectively, when theimaging device acquires the image. The processor is configured toidentify a position on the three-dimensional model corresponding to apixel of the image on the basis of the estimated position and theposture. The processor is configured to determine whether or not theimaging condition that has been set in the first region or the secondregion including the position on the three-dimensional model issatisfied. The processor is configured to display observationinformation on the display on the basis of a result of determination ofthe imaging condition. The observation information represents whether ornot the first region or the second region including the position on thethree-dimensional model has been observed.

According to a seventeenth aspect of the present invention, anon-transitory computer-readable recording medium saves a program forcausing a computer to execute an image display step, an estimation step,an identification step, a determination step, and an observationinformation display step. The observation device includes an insertionunit and a storage medium. The insertion unit includes an imaging deviceconfigured to acquire an image of a subject inside an observation objectand is configured to be inserted into the observation object. Thestorage medium is configured to store an imaging condition that has beenset in each of a first region and a second region. The first region ispart of a three-dimensional model of the subject. The second region isother than the first region in the three-dimensional model. The imagingcondition of the first region and the imaging condition of the secondregion are different from each other. The computer displays the image ona display in the image display step. The computer estimates, in theestimation step, a position of the imaging device and a posture of theimaging device. The position and the posture are a position and aposture, respectively, when the imaging device acquires the image. Thecomputer identifies, in the identification step, a position on thethree-dimensional model corresponding to a pixel of the image on thebasis of the position and the posture estimated in the estimation step.The computer determines, in the determination step, whether or not theimaging condition that has been set in the first region or the secondregion including the position on the three-dimensional model issatisfied. The computer displays, in the observation information displaystep, observation information on the display on the basis of a result ofthe determination step, the observation information representing whetheror not the first region or the second region including the position onthe three-dimensional model has been observed.

According to an eighteenth aspect of the present invention, in theseventeenth aspect, the program may cause the computer to furtherexecute a region setting step in which the computer sets the firstregion and the second region to the three-dimensional model.

According to a nineteenth aspect of the present invention, in theseventeenth aspect, the program may cause the computer to furtherexecute a feature point detection step, an association step, acalculation step, and a generation step. The computer detects a featurepoint in each of a plurality of images acquired by the imaging device inthe feature point detection step. The computer associates the featurepoint between the images included in the plurality of images in theassociation step. The computer calculates a position of the imagingdevice and a posture of the imaging device on the basis of the featurepoint in the calculation step. The computer generates thethree-dimensional model on the basis of the position and the posturecalculated in the calculation step in the generation step. Before theregion setting step is executed, the feature point detection step, theassociation step, the calculation step, and the generation step may beexecuted.

According to a twentieth aspect of the present invention, in theseventeenth aspect, the computer may estimate the position and theposture on the basis of at least the image and the three-dimensionalmodel in the estimation step.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing an entire configuration of anendoscope device according to a first embodiment of the presentinvention.

FIG. 2 is a block diagram showing an internal configuration of theendoscope device according to the first embodiment of the presentinvention.

FIG. 3 is a block diagram showing a functional configuration of a PCaccording to the first embodiment of the present invention.

FIG. 4 is a block diagram showing a functional configuration of a CPUaccording to the first embodiment of the present invention.

FIG. 5 is a flow chart showing a procedure of processing executed by thePC according to the first embodiment of the present invention.

FIG. 6 is a diagram showing an index of determination and a thresholdvalue of determination in the first embodiment of the present invention.

FIG. 7 is a diagram showing an inspection image and a motion vector inthe first embodiment of the present invention.

FIG. 8 is a diagram showing the relationship between a motion vector anda threshold value in the first embodiment of the present invention.

FIG. 9 is a schematic diagram showing a situation in which an image isacquired in the first embodiment of the present invention.

FIG. 10 is a flow chart showing a procedure of processing for generatinga three-dimensional model in the first embodiment of the presentinvention.

FIG. 11 is a flow chart showing a procedure of processing executed bythe CPU according to the first embodiment of the present invention.

FIG. 12 is a flow chart showing a procedure of processing executed bythe CPU according to the first embodiment of the present invention.

FIG. 13 is a diagram showing an example of an image displayed on adisplay unit according to the first embodiment of the present invention.

FIG. 14 is a diagram showing an example of an image displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 15 is a diagram showing a method of identifying a position on athree-dimensional model in the first embodiment of the presentinvention.

FIG. 16 is a diagram showing an example of an image displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 17 is a diagram showing an example of an image displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 18 is a block diagram showing a functional configuration of a PCaccording to a second modified example of the first embodiment of thepresent invention.

FIG. 19 is a flow chart showing a procedure of processing executed bythe PC according to the second modified example of the first embodimentof the present invention.

FIG. 20 is a block diagram showing a functional configuration of a CPUaccording to a second embodiment of the present invention.

FIG. 21 is a flow chart showing a procedure of processing executed bythe CPU according to the second embodiment of the present invention.

FIG. 22 is a diagram showing a method of notifying a user of a cause ofdetermination that a position on a three-dimensional model has not beenobserved in the second embodiment of the present invention.

FIG. 23 is a diagram showing a method of detecting a non-observedposition behind a visual field of an imaging device in the secondembodiment of the present invention.

FIG. 24 is a flow chart showing a procedure of processing executed by aCPU according to a third embodiment of the present invention.

FIG. 25 is a flow chart showing a procedure of processing executed bythe CPU according to the third embodiment of the present invention.

FIG. 26 is a diagram showing an example of an image displayed on adisplay unit according to the third embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, embodiments of the present invention will be described withreference to the drawings. Hereinafter, an example in which anobservation device is an endoscope device will be described. Theobservation device has only to be a device including an insertion unitto be inserted into an observation object and acquiring an image of asubject in the observation object. The observation device is not limitedto an endoscope device.

First Embodiment

FIG. 1 shows an entire configuration of an endoscope device 1 accordingto a first embodiment of the present invention. FIG. 2 shows an internalconfiguration of the endoscope device 1. The endoscope device 1 images asubject and generates an image. In order to observe various subjects, aninspector can perform replacement of an optical adaptor mounted at a tipend of an insertion unit 2, selection of a built-in video processingprogram, and addition of a video processing program. Hereinafter, a casein which a user designates a region of interest prior to inspection willbe described. A region of interest is defined as an inspection regionfor which a condition for determining whether or not the inspectionregion has been observed is different from the condition of otherinspection regions.

The endoscope device 1 shown in FIG. 1 includes the insertion unit 2, amain body unit 3, an operation unit 4, and a display unit 5.

The insertion unit 2 is inserted into the inside of a subject. Theinsertion unit 2 has a long and thin bendable tube shape from the tipend 20 to a base end portion. The insertion unit 2 images a subject andoutputs an imaging signal to the main body unit 3. An optical adapter ismounted on the tip end 20 of the insertion unit 2. For example, asingle-eye optical adapter is mounted on the tip end 20 of the insertionunit 2. The main body unit 3 is a control device including a housingunit that houses the insertion unit 2. The operation unit 4 accepts auser's operation for the endoscope device 1. The display unit 5 includesa display screen and displays an image of a subject acquired by theinsertion unit 2, an operation menu, and the like on the display screen.

The operation unit 4 is a user interface. For example, the operationunit 4 is at least one of a button, a switch, a key, a mouse, ajoystick, a touch pad, a track ball, and a touch panel. The display unit5 is a monitor (display) such as a liquid crystal display (LCD). Thedisplay unit 5 may be a touch panel. In such a case, the operation unit4 and the display unit 5 are integrated.

The main body unit 3 shown in FIG. 2 includes an endoscope unit 8, acamera control unit (CCU) 9, and a control device 10. The endoscope unit8 includes a light source device and a bending device not shown in thedrawing. The light source device supplies illumination light that isnecessary for observation. The bending device bends a bending mechanismthat is built into the insertion unit 2. An imaging device 28 is builtinto the tip end 20 of the insertion unit 2. The imaging device 28 is animage sensor. The imaging device 28 photo-electrically converts anoptical image of a subject formed by an optical adaptor and generates animaging signal. The CCU 9 drives the imaging device 28. An imagingsignal output from the imaging device 28 is input to the CCU 9. The CCU9 executes a pre-process including amplification, noise elimination, andthe like for the imaging signal acquired by the imaging device 28. TheCCU 9 converts the imaging signal for which the pre-process has beenexecuted into a video signal such as an NTSC signal.

The control device 10 includes a video signal processing circuit 12, aread only memory (ROM) 13, a random access memory (RAM) 14, a cardinterface 15, an external device interface 16, a control interface 17,and a central processing unit (CPU) 18 a.

The video signal processing circuit 12 performs predetermined videoprocessing on the video signal output from the CCU 9. For example, thevideo signal processing circuit 12 performs video processing related toimprovement of visibility. For example, the video processing is colorreproduction, gray scale correction, noise suppression, contourenhancement, and the like. For example, the video signal processingcircuit 12 combines the video signal output from the CCU 9 and a graphicimage signal generated by the CPU 18 a. The graphic image signalincludes an image of the operation screen and the like. The video signalprocessing circuit 12 outputs a combined video signal to the displayunit 5.

The ROM 13 is a nonvolatile recording medium on which a program for theCPU 18 a to control the operation of the endoscope device 1 is recorded.The RAM 14 is a volatile recording medium that temporarily storesinformation used by the CPU 18 a for controlling the endoscope device 1.The CPU 18 a controls the operation of the endoscope device 1 on thebasis of a program recorded on the ROM 13.

A memory card 42 that is a removable recording medium is connected tothe card interface 15. The card interface 15 inputs control processinginformation, image information, and the like stored in the memory card42 to the control device 10. In addition, the card interface 15 recordsthe control processing information, the image information, and the likegenerated by the endoscope device 1 on the memory card 42.

An external device such as a USB device is connected to the externaldevice interface 16. For example, a personal computer (PC) 41 isconnected to the external device interface 16. The external deviceinterface 16 transmits information to the PC 41 and receives informationfrom the PC 41. Accordingly, a monitor of the PC 41 can displayinformation. In addition, by inputting an instruction to the PC 41, auser can perform an operation related to control of the endoscope device1.

The control interface 17 performs communication with the operation unit4, the endoscope unit 8, and the CCU 9 for operation control. Thecontrol interface 17 notifies the CPU 18 a of an instruction input tothe operation unit 4 by a user. The control interface 17 outputs controlsignals used for controlling the light source device and the bendingdevice to the endoscope unit 8. The control interface 17 outputs acontrol signal used for controlling the imaging device 28 to the CCU 9.

A program executed by the CPU 18 a may be recorded on acomputer-readable recording medium. The program recorded on thisrecording medium may be read and executed by a computer other than theendoscope device 1. For example, the PC 41 may read and execute theprogram. The PC 41 may control the endoscope device 1 by transmittingcontrol information used for controlling the endoscope device 1 to theendoscope device 1 in accordance with a program. Alternatively, the PC41 may acquire a video signal from the endoscope device 1 and mayprocess the acquired video signal.

The program described above may be transmitted from the computer storingthe program to the endoscope device 1 through a transmission medium ortransmission waves in a transmission medium. The “transmission medium”transmitting the program is a medium having a function of transmittinginformation. The medium having the function of transmitting informationincludes a network (communication network) such as the Internet and acommunication circuit line (communication line) such as a telephoneline. The program described above may realize some of the functionsdescribed above. In addition, the program described above may be adifferential file (differential program). A combination of a programthat has already been recorded in a computer and a differential programmay realize the functions described above.

The above-described endoscope device 1 includes the insertion unit 2,the RAM 14, and the CPU 18 a. The insertion unit 2 includes the imagingdevice 28 and is inserted into an inspection object (observationobject). The imaging device 28 images a subject in the inspection objectand generates an imaging signal. In this way, the imaging device 28generates an image (image data) of the subject on the basis of anoptical image of the subject in an imaging visual field. The imagingdevice 28 generates an inspection moving image of the subject bycontinuously generating a plurality of images. The image generated bythe imaging device 28 is input to the CPU 18 a through the video signalprocessing circuit 12. The RAM 14 (storage medium) stores an imagingcondition that has been set in each of a region of interest (firstregion) that is part of a three-dimensional (3D) model of the subjectand a general region (second region) other than the region of interestin the 3D model. The imaging condition is used for determining whetheror not a region of the subject has been observed. The imaging conditionof the region of interest and the imaging condition of the generalregion are different from each other. Hereinafter, an example in whichthe PC 41 performs setting of a region and setting of an imagingcondition will be described.

FIG. 3 shows a functional configuration of the PC 41. The PC 41 shown inFIG. 3 includes a CPU 43, an operation unit 44, a display unit 45, acommunication unit 46, and a memory 47.

The CPU 43 controls an operation of the PC 41. The operation unit 44 isa user interface. The operation unit 44 accepts a user's operation forthe PC 41. A user can input various kinds of information to the PC 41 byoperating the operation unit 44. The operation unit 44 acceptsinformation input by a user and outputs the information to the CPU 43.The display unit 45 includes a display screen and displays a 3D model ofa subject in an inspection object and the like on the display screen.The display unit 45 is a monitor (display) such as an LCD. The displayunit 45 may be a touch panel. In such a case, the operation unit 44 andthe display unit 45 are integrated. The communication unit 46 performscommunication with the endoscope device 1. The memory 47 stores aninspection moving image acquired by the endoscope device 1 and the like.

The functions of the CPU 43 are constituted by a video selection unit430, a condition acceptance unit 431, a 3D model generation unit 432, adisplay control unit 433, a region acceptance unit 434, an indexacceptance unit 435, a threshold value acceptance unit 436, and acondition setting unit 437. At least one of the blocks in the CPU 43shown in FIG. 3 may be constituted by a circuit different from the CPU43.

Each unit in the CPU 43 may be constituted by at least one of aprocessor and a logic circuit. For example, the processor is at leastone of a CPU, a digital signal processor (DSP), and a graphicsprocessing unit (GPU). For example, the logic circuit is at least one ofan application specific integrated circuit (ASIC) and afield-programmable gate array (FPGA). Each unit in the CPU 43 mayinclude one or a plurality of processors. Each unit in the CPU 43 mayinclude one or a plurality of logic circuits.

The communication unit 46 receives a plurality of inspection movingimages from the endoscope device 1. The memory 47 stores the receivedplurality of inspection moving images. A user inputs information thatrepresents an inspection moving image to be used for setting a region ofinterest and setting an imaging condition to the operation unit 44. Thevideo selection unit 430 selects one of the plurality of inspectionmoving images on the basis of the information input to the operationunit 44 by a user. The memory 47 may store only one inspection movingimage and the video selection unit 430 may select the inspection movingimage. Hereinafter, an example in which a moving image of a subject isused for setting a region of interest and setting an imaging conditionwill be described. A moving image is not necessarily used and a stillimage may be used.

A user inputs information that represents a condition for generating a3D model of a subject to the operation unit 44. The condition acceptanceunit 431 accepts the condition for generating the 3D model of thesubject on the basis of the information input to the operation unit 44by a user. Specifically, the condition includes an internal parameter ofa camera, a distortion correction parameter of the camera, a settingvalue, a reference length, and the like. The setting value is used for avariety of pieces of processing for generating a 3D model. The referencelength is used for matching the 3D model with the actual scale of asubject. The information accepted by the condition acceptance unit 431is stored on the memory 47.

The 3D model generation unit 432 (processing unit) generates(reconfigures) a 3D model of a subject. The 3D model generated by the 3Dmodel generation unit 432 is stored on the memory 47.

The display control unit 433 displays the 3D model generated by the 3Dmodel generation unit 432 on the display unit 45.

A user inputs information that represents a region of interest on asubject to the operation unit 44. The region acceptance unit 434 (regionsetting unit) accepts the region of interest on the basis of theinformation input to the operation unit 44 by a user. A region that isnot designated as a region of interest by a user is accepted as ageneral region. The region acceptance unit 434 sets the region ofinterest and the general region to the 3D model generated by the 3Dmodel generation unit 432. Each region accepted by the region acceptanceunit 434 is stored on the memory 47.

A user inputs information that represents an index for determiningwhether or not a region on a subject has been observed to the operationunit 44. The index represents a type of an imaging condition. It ispossible for a user to designate an index for each region. The indexacceptance unit 435 accepts an index on the basis of the informationinput to the operation unit 44 by a user. The index accepted by theindex acceptance unit 435 is stored on the memory 47.

A user inputs information that represents a threshold value fordetermining whether or not a region on a subject has been observed tothe operation unit 44. It is possible for a user to designate athreshold value for each region. The threshold value acceptance unit 436accepts a threshold value on the basis of the information input to theoperation unit 44 by a user. The threshold value accepted by thethreshold value acceptance unit 436 is stored on the memory 47.

The condition setting unit 437 (imaging condition setting unit) sets animaging condition including an index and a threshold value to each ofthe region of interest and the general region of the 3D model generatedby the 3D model generation unit 432. The condition setting unit 437 setsan index of the region of interest and a threshold value of the regionof interest to the region of interest. The condition setting unit 437sets an index of the general region and a threshold value of the generalregion to the general region. The condition setting unit 437 generates a3D model to which an imaging condition of each region has been set. The3D model includes data of three-dimensional coordinates of a subject,information of the region of interest, information of the generalregion, and the imaging condition of each region. The 3D model generatedby the condition setting unit 437 is stored on the memory 47. Thecommunication unit 46 transmits the 3D model to which the imagingcondition has been set to the endoscope device 1.

The external device interface 16 of the endoscope device 1 transmits theinspection moving image acquired by the imaging device 28 to the PC 41.The external device interface 16 receives the 3D model to which theimaging condition has been set from the PC 41. The 3D model is stored onthe RAM 14.

FIG. 4 shows a functional configuration of the CPU 18 a. The functionsof the CPU 18 a are constituted by a main control unit 180, an imageacquisition unit 181, a display control unit 182, a 3D model selectionunit 183, a camera position-posture estimation unit 184, a positionidentification unit 185, and a determination unit 186. At least one ofthe blocks shown in FIG. 4 may be constituted by a circuit other thanthe CPU 18 a.

Each unit shown in FIG. 4 may be constituted by at least one of aprocessor and a logic circuit. Each unit shown in FIG. 4 may include oneor a plurality of processors. Each unit shown in FIG. 4 may include oneor a plurality of logic circuits.

The main control unit 180 controls processing executed by each unit. Theimage acquisition unit 181 acquires an image generated by the imagingdevice 28 from the video signal processing circuit 12. The acquiredimage is held on the RAM 14.

The display control unit 182 displays the image generated by the imagingdevice 28 on the display unit 5. For example, the display control unit182 controls the processing executed by the video signal processingcircuit 12. The display control unit 182 causes the video signalprocessing circuit 12 to output the processed image to the display unit5. The display unit 5 displays the image output from the video signalprocessing circuit 12.

The display control unit 182 displays a 3D model, a determinationresult, and the like on the display unit 5. In other words, the displaycontrol unit 182 displays various kinds of information on an image. Forexample, the display control unit 182 generates a graphic image signalof various kinds of information. The display control unit 182 outputsthe generated graphic image signal to the video signal processingcircuit 12. The video signal processing circuit 12 combines a videosignal output from the CCU 9 and the graphic image signal output fromthe CPU 18 a. In this way, various kinds of information are superimposedon an image. The video signal processing circuit 12 outputs the combinedvideo signal to the display unit 5. The display unit 5 displays an imageon which various kinds of information are superimposed on the basis ofthe video signal.

A user inputs information that represents a 3D model of a subject to theoperation unit 4. The operation unit 4 accepts the information input tothe operation unit 4 by a user and outputs the information. Theinformation input to the operation unit 4 is input to the controlinterface 17 that is an input unit. The information input to the controlinterface 17 is input to the CPU 18 a. The 3D model selection unit 183selects a 3D model on the basis of the information input to theoperation unit 4 by a user.

The camera position-posture estimation unit 184 estimates the positionof the imaging device 28 and the posture of the imaging device 28. Theestimated position and posture are a position of the imaging device 28and a posture of the imaging device 28, respectively, when the imagingdevice 28 acquires an image. The camera position-posture estimation unit184 uses the image acquired by the image acquisition unit 181 and the 3Dmodel selected by the 3D model selection unit 183 in order to estimatethe position of the imaging device 28 and the posture of the imagingdevice 28.

The position identification unit 185 identifies a position on the 3Dmodel corresponding to a pixel of the image acquired by the imageacquisition unit 181. The position identification unit 185 uses theimage acquired by the image acquisition unit 181, the 3D model selectedby the 3D model selection unit 183, and the position and the postureestimated by the camera position-posture estimation unit 184 in order toidentify a position on the 3D model.

The determination unit 186 determines whether or not the imagingcondition that has been set to the region of interest or the generalregion including the position identified by the position identificationunit 185 is satisfied. Specifically, the determination unit 186 acquiresthe index and the threshold value that has been set to the positionidentified by the position identification unit 185. The determinationunit 186 calculates an index value on the basis of the image acquired bythe image acquisition unit 181 and the position and the postureestimated by the camera position-posture estimation unit 184. Thedetermination unit 186 determines whether or not the imaging conditionis satisfied by comparing the calculated index value with the thresholdvalue.

The display control unit 182 displays observation information on thedisplay unit 5 on the basis of a result of the determination performedby the determination unit 186. The observation information representswhether or not the region of interest or the general region includingthe position identified by the position identification unit 185 has beenobserved. The display control unit 182 generates a graphic image signalof the observation information. Thereafter, processing similar to theprocessing for displaying various kinds of information is executed.

Characteristic processing of the first embodiment includes processingexecuted prior to inspection and processing executed during inspection.Hereinafter, each of the pieces of processing will be described.Hereinafter, an image generated by the imaging device 28 for inspectionis described as an inspection image.

A procedure of processing executed prior to inspection will be describedwith reference to FIG. 5. FIG. 5 shows a procedure of processingexecuted by the PC 41. The PC 41 may be any of a desktop and a laptop.In addition, the processing shown in FIG. 5 may be executed in a cloudenvironment.

A user inputs information that represents an inspection moving imageused for setting a region of interest and setting an imaging conditionto the operation unit 44. The video selection unit 430 selects one of aplurality of inspection moving images on the basis of the informationinput to the operation unit 44 by a user (Step S101).

After Step S101, a user inputs information that represents a conditionfor generating a 3D model of a subject to the operation unit 44. Thecondition acceptance unit 431 accepts the condition for generating the3D model of the subject on the basis of the information input to theoperation unit 44 by a user (Step S102). As described above, thecondition includes an internal parameter of a camera, a distortioncorrection parameter of the camera, a setting value, a reference length,and the like. A user does not need to designate all of these conditions.The CPU 43 may automatically set at least one of these conditions.

After Step S102, the 3D model generation unit 432 generates a 3D modelof a subject on the basis of the selected inspection moving image andthe condition for generating the 3D model (Step S103). A procedure ofspecific processing executed by the 3D model generation unit 432 will bedescribed later.

After Step S103, the display control unit 433 displays the generated 3Dmodel on the display unit 45 (Step S104).

After Step S104, a user determines a region to be designated as a regionof interest in the 3D model and inputs information that represents theregion to the operation unit 44. The region acceptance unit 434 acceptsthe region of interest on the basis of the information input to theoperation unit 44 by a user. The region acceptance unit 434 sets theregion of interest and a general region to the generated 3D model (StepS105).

For example, a user designates a region in which the degree ofimportance of inspection is high as a region of interest. A region ofinterest may be a region in which defects were frequently found in thepast inspection. A region of interest may be a region for which it wasdetermined that follow-up observation was necessary in the pastinspection. A region of interest may be a region in which a flaw islikely to occur due to a design.

A region of interest may be a region in which the degree of importanceis not especially high. For example, a region of interest may be aregion required to be inspected under a specific condition. The specificcondition is an object distance, composition of an image, or a type ofan optical adaptor. A region of interest may be set on the basis of aninspection manual or design intention of an inspection object. Forexample, in inspection of aircraft engines, design intention is sharedbetween an engine manufacturer and an inspection department of eachbusiness in many cases.

In the first embodiment, a user designates a region of interest on a 3Dmodel while the 3D model is displayed on the display unit 45. A userdoes not need to directly designate a region on a 3D model as a regionof interest. A user may designate a region on a two-dimensional subjectseen in an inspection moving image as a region of interest.Specifically, in Step S104, an inspection moving image is displayed onthe display unit 45 instead of a 3D model. A user determines a region tobe designated as a region of interest in a two-dimensional subject andinputs information that represents the region to the operation unit 44.The region acceptance unit 434 accepts the region of interest on thebasis of the information input to the operation unit 44 by a user. Aposition on a 3D model and a position on a two-dimensional subject areassociated with each other. Therefore, it is possible for the CPU 43 toidentify a region on a 3D model corresponding to a region designated ona two-dimensional subject.

After Step S105, a user inputs information that represents an index ofdetermination and a threshold value of determination to the operationunit 44. The index acceptance unit 435 accepts the index on the basis ofthe information input to the operation unit 44 by a user. The thresholdvalue acceptance unit 436 accepts the threshold value on the basis ofthe information input to the operation unit 44 by a user (Step S106).The index and the threshold value are accepted for each region. In otherwords, the index and the threshold value of the region of interest areaccepted and the index and the threshold value of the general region areaccepted.

FIG. 6 shows an example of an index of determination and a thresholdvalue of determination. Four indices are shown in FIG. 6. The fourindices include an index A (object distance), an index B (observationtime), an index C (image luminance), and an index D (position within avisual field). The index A represents the distance from a subject to thetip end of the endoscope. The index B represents a length of time inwhich the tip end of the endoscope is stopped for observation. The indexC represents a luminance value of an inspection image. The index Drepresents a field angle from the center of an image to the position ofeach region. Each of the indices shown in FIG. 6 is calculated on thebasis of an inspection image. A threshold value of each index in each oftwo regions of interest and one general region are shown in FIG. 6. Thetwo regions of interest are a region R1 of interest and a region R2 ofinterest.

In the example shown in FIG. 6, all of the four indices are set to thetwo regions of interest. In the example shown in FIG. 6, only the indexA and the index C are set to the general region.

An index set in each region is not limited to the example shown in FIG.6. For example, the index B may not be included. An index E that is notshown in FIG. 6 may be added. At least one index has only to be set. Anindex may be the reliability of three-dimensional coordinates calculatedon the basis of an inspection image. An index may be a relative posture(observation angle) of the tip end of the endoscope with respect to asubject. An index may be a type of an optical adaptor used forinspection.

A threshold value set to each region is not limited to the example shownin FIG. 6. A threshold value may be changed in accordance with anendoscope equipment or a subject. A threshold value may be changed inaccordance with the proficiency of a user for inspection.

Specific determination processing using the index B will be describedwith reference to FIG. 7 and FIG. 8. Three inspection images I11 to I13shown in FIG. 7 are defined as described below. The inspection image I11(reference image) is an image acquired at a first time point by theimage acquisition unit 181. The inspection image I12 is an imageacquired at a second time point following the first time point by theimage acquisition unit 181. The inspection image I13 is an imageacquired at a third time point following the second time point by theimage acquisition unit 181.

The determination unit 186 detects a representative point seen in theinspection image I11. The determination unit 186 detects a motion vectorV12 of the representative point on the basis of the inspection image I11and the inspection image I12. Similarly, the determination unit 186detects a motion vector V13 of the representative point on the basis ofthe inspection image I12 and the inspection image I13.

FIG. 8 shows the relationship between each motion vector and a thresholdvalue. A threshold value TH1 is an allowable value of motion determinedin advance. In this example, the determination unit 186 calculates thesum of motion vectors between timings at which respective inspectionimages of these three frames are acquired. If the sum does not exceedthe threshold value TH1, the determination unit 186 determines that asubject has been inspected in a stationary state. In this example, thesum (maximum distance) of motion vectors between respective inspectionimages is set as the threshold value. A value that is allowable as theamount of motion between respective inspection images may be set as thethreshold value. In other words, the determination unit 186 maydetermine whether or not the amount of motion between respectiveinspection images in a period set in advance is less than or equal tothe threshold value.

In a case in which the motion of a camera is translational motion, theamount of motion in an inspection image varies for each object distance.For this reason, after the index A (object distance) is satisfied, thedetermination unit 186 may determine the value of the index B(observation time).

After Step S106, the condition setting unit 437 sets an imagingcondition of each of the region of interest and the general region tothe generated 3D model. In this way, the condition setting unit 437generates a 3D model to which the imaging condition has been set (StepS107). When Step S107 is executed, the processing shown in FIG. 5 iscompleted.

After the processing shown in FIG. 5 is completed, the communicationunit 46 transmits the 3D model to which the imaging condition has beenset to the endoscope device 1. The external device interface 16 of theendoscope device 1 receives the 3D model from the PC 41. The received 3Dmodel is stored on the RAM 14.

A procedure of specific processing executed by the 3D model generationunit 432 will be described. The 3D model generation unit 432 uses theinspection moving image selected by the video selection unit 430 and thecondition accepted by the condition acceptance unit 431. Hereinafter, anexample in which the 3D model generation unit 432 uses two inspectionimages extracted from the inspection moving image will be described.When a camera captures two inspection images, two viewpoints of thecamera are different from each other. Even when three or more inspectionimages are used, a basic principle is not changed from that of the casein which two inspection images are used. A method described below may beapplied also to a case in which three or more inspection images areused.

FIG. 9 schematically shows a status of image acquisition in a case inwhich two inspection images of a subject are acquired. In thedescription described below, an expression “camera” in a broad sense isused. The camera in the description described below specificallyindicates an observation optical system of the tip end of the endoscope(the tip end 20 of the insertion unit 2).

As shown in FIG. 9, first, an inspection image I₁ is acquired in animaging state c₁ of the camera. Next, an inspection image I₂ is acquiredin an imaging state c₂ of the camera. At least one of an imagingposition and an imaging posture is different between the imaging statec₁ and the imaging state c₂. In FIG. 9, both the imaging position andthe imaging posture are different between the imaging state c₁ and theimaging state c₂.

In each embodiment of the present invention, it is assumed that theinspection image I₁ and the inspection image I₂ are acquired by the sameendoscope. In addition, in each embodiment of the present invention, itis assumed that parameters of an objective optical system of theendoscope do not change. The parameters of the objective optical systemare a focal distance, a distortion aberration, a pixel size of an imagesensor, and the like. Hereinafter, for the convenience of description,the parameters of the objective optical system will be abbreviated tointernal parameters. When such conditions are assumed, the internalparameters describing characteristics of the optical system of theendoscope can be used in common regardless of the position and theposture of the tip end of the endoscope. In each embodiment of thepresent invention, it is assumed that the internal parameters areacquired at the time of factory shipment. In addition, it is assumedthat the internal parameters are known at the time of acquiring aninspection image.

In each embodiment of the present invention, it is assumed that two ormore inspection images are extracted from the inspection moving imageand the inspection moving image is acquired by one endoscope. However,the present invention is not limited to this. For example, the presentinvention may be applied to also a case in which a 3D model is restoredby using a plurality of inspection moving images acquired by a pluralityof endoscopes. In this case, the inspection image I₁ and the inspectionimage I₂ have only to be acquired by using different endoscopes and eachinternal parameter has only to be stored for each endoscope. Even if theinternal parameters are unknown, it is possible to perform calculationby using the internal parameters as variables. For this reason, thesubsequent procedure does not greatly change in accordance with whetheror not the internal parameters are known.

A procedure for calculating three-dimensional coordinates of a subjecton the basis of acquired two inspection images will be described withreference to FIG. 10. FIG. 10 shows the procedure of processing forgenerating a 3D model.

First, the 3D model generation unit 432 executes a feature pointdetection processing (Step S103 a). The 3D model generation unit 432detects a feature point of each of acquired two inspection images in thefeature point detection processing. The feature point represents acorner, an edge, and the like in which an image luminance gradient islarge in information of a subject seen in the inspection image. As amethod of detecting this feature point, a scale-invariant featuretransform (SIFT), a feature from accelerated segment test (FAST), or thelike is used. By using such a method, a feature point within an imagecan be detected.

FIG. 9 shows an example in which a feature point m₁ is detected from theinspection image I₁ and a feature point m₂ is detected from theinspection image I₂. Although only one feature point of each image isshown in FIG. 9, in fact, a plurality of feature points are detected ineach image. There is a possibility that the number of feature pointsdetected in each image is different. Each feature point detected fromeach image is converted into data called a feature quantity. The featurequantity is data that represent a feature of a feature point.

After Step S103 a, the 3D model generation unit 432 executes a featurepoint associating processing (Step S103 b). In the feature pointassociating processing, the 3D model generation unit 432 comparescorrelations of feature quantities between inspection images for eachfeature point detected in the feature point detection processing (StepS103 a). In a case in which the correlations of the feature quantitiesare compared and a feature point of which feature quantities are closeto those of a feature point of another inspection image is found in eachinspection image, the 3D model generation unit 432 stores theinformation on the RAM 14. In this way, the 3D model generation unit 432associates a feature point of each inspection image together. On theother hand, in a case in which a feature point of which featurequantities are close to those of a feature point of another inspectionimage is not found, the 3D model generation unit 432 discardsinformation of the feature point.

After Step S103 b, the 3D model generation unit 432 reads coordinates offeature points of two inspection images associated with each other (afeature point pair) from the RAM 14. The 3D model generation unit 432executes processing of calculating a position and a posture on the basisof the read coordinates (Step S103 c). In the processing of calculatinga position and a posture, the 3D model generation unit 432 calculates arelative position and a relative posture between the imaging state c₁ ofthe camera that has acquired the inspection image I₁ and the imagingstate c₂ of the camera that has acquired the inspection image I₂. Morespecifically, the 3D model generation unit 432 calculates a matrix E bysolving the following Equation (1) using an epipolar restriction.

$\begin{matrix}{{p_{1}^{T}{Ep}_{2}} = {{0\mspace{14mu} E} = {{{\lbrack t\rbrack_{X}R}\mspace{14mu}\because\lbrack t\rbrack_{X}} = \begin{pmatrix}0 & {- t_{z}} & t_{y} \\t_{z} & 0 & {- t_{x}} \\{- t_{y}} & t_{x} & 0\end{pmatrix}}}} & (1)\end{matrix}$

The matrix E is called a basic matrix. The basic matrix E is a matrixstoring a relative position and a relative posture between the imagingstate c₁ of the camera that has acquired the inspection image I₁ and theimaging state c₂ of the camera that has acquired the inspection imageI₂. In Equation (1), a matrix p₁ is a matrix including coordinates of afeature point detected from the inspection image I₁. A matrix p₂ is amatrix including coordinates of a feature point detected from theinspection image I₂. The basic matrix E includes information related toa relative position and a relative posture of the camera and thuscorresponds to external parameters of the camera. The basic matrix E canbe solved by using a known algorithm.

As shown in FIG. 9, Expression (2) and Expression (3) are satisfied in acase in which the amount of position change of the camera is t and theamount of posture change of the camera is R.

$\begin{matrix}{t = \left( {t_{x},t_{y},t_{z}} \right)} & (2) \\\begin{matrix}{R =} & {{R_{x}(\alpha)}{R_{y}(\beta)}{R_{z}(\gamma)}} \\{=} & {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\mspace{14mu}\alpha} & {{- \sin}\mspace{14mu}\alpha} \\0 & {\sin\mspace{14mu}\alpha} & {\cos\mspace{14mu}\alpha}\end{pmatrix}\begin{pmatrix}{\cos\mspace{14mu}\beta} & 0 & {\sin\mspace{14mu}\beta} \\0 & 1 & 0 \\{{- \sin}\mspace{14mu}\beta} & 0 & {\cos\mspace{14mu}\beta}\end{pmatrix}} \\ & {\begin{pmatrix}{\cos\mspace{14mu}\gamma} & {{- \sin}\mspace{11mu}\gamma} & 0 \\{\sin\mspace{14mu}\gamma} & {\cos\mspace{14mu}\gamma} & 0 \\0 & 0 & 1\end{pmatrix}}\end{matrix} & (3)\end{matrix}$

In Expression (2), the amount of movement in an x-axis direction isexpressed as t_(x), the amount of movement in a y-axis direction isexpressed as t_(y), and the amount of movement in a z-axis direction isexpressed as t_(z). In Expression (3), a rotation amount α around thex-axis is expressed as R_(x)(α), a rotation amount β around the y axisis expressed as R_(y)(β), and a rotation amount γ around the z axis isexpressed as R_(z)(γ). After the basic matrix E is calculated,optimization processing called bundle adjustment may be executed inorder to improve restoration accuracy of three-dimensional coordinates.

After Step S103 c, the 3D model generation unit 432 executes processingof restoring a three-dimensional shape of a subject on the basis of therelative position and the relative posture of the camera (the amount ofposition change t and the amount of posture change R) calculated in StepS103 c (Step S103 d). The 3D model generation unit 432 generates a 3Dmodel of a subject in the processing of restoring a three-dimensionalshape. As a technique for restoring a three-dimensional shape of asubject, there is matching processing that uses patch-based multi-viewstereo (PMVS) and parallelization stereo and the like. However, a meanstherefor is not particularly limited.

After Step S103 d, the 3D model generation unit 432 executes athree-dimensional coordinate transforming processing on the basis of thethree-dimensional shape data of a subject calculated in the processingof restoring a three-dimensional shape (Step S103 d) and the referencelength accepted by the condition acceptance unit 431 (Step S103 e). The3D model generation unit 432 transforms the three-dimensional shape dataof a subject into three-dimensional coordinate data having a dimensionof a length in the three-dimensional coordinate transforming processing.When Step S103 e is executed, the processing shown in FIG. 10 iscompleted.

A procedure of processing executed during inspection will be describedwith reference to FIG. 11 and FIG. 12. FIG. 11 and FIG. 12 show aprocedure of processing executed by the CPU 18 a.

After the power of the endoscope device 1 is supplied, the insertionunit 2 is inserted into an inspection object and inspection is started.After the inspection is started, a user inputs information thatrepresents a 3D model of a subject to the operation unit 4. Theoperation unit 4 accepts the information input to the operation unit 4by a user. The 3D model selection unit 183 selects a 3D model stored onthe RAM 14 on the basis of the information input to the operation unit 4by a user. The 3D model selection unit 183 reads the selected 3D modelfrom the RAM 14 (Step S201).

After Step S201, the display control unit 182 generates a graphic imagesignal for displaying the 3D model selected by the 3D model selectionunit 183 and outputs the graphic image signal to the video signalprocessing circuit 12. The video signal processing circuit 12 combinesthe graphic image signal and a video signal and outputs the combinedvideo signal to the display unit 5. The video signal includes color dataof each pixel in an image of the 3D model. Each pixel is associated withthree-dimensional coordinates. The display unit 5 displays an image ofthe 3D model on the basis of the video signal (Step S202).

FIG. 13 shows an example of an image displayed on the display unit 5 inStep S202. A 3D model M11 is displayed on the display unit 5. A regionR11 of interest and a region R12 of interest are displayed on the 3Dmodel M11. A general region is a region on the 3D model M11 excludingthe region R11 of interest and the region R12 of interest. For example,the region R11 of interest and the region R12 of interest are displayedin a predetermined color. The region R11 of interest and the region R12of interest may be translucently displayed such that a user can visuallyrecognize the 3D model M11. For example, the region R11 of interestcorresponds to the region R1 of interest in FIG. 6 and the region R12 ofinterest corresponds to the region R2 of interest in FIG. 6. As long asa user can distinguish between the region of interest and the generalregion, any display method may be used.

After Step S202, the image acquisition unit 181 acquires an inspectionimage from the video signal processing circuit 12 (Step S203). AfterStep S203, the display control unit 182 displays the acquired inspectionimage on the display unit 5 (Step S204).

A region of interest may be displayed on the inspection image. In a casein which a small region of interest such as a cooling hole of anaircraft engine is far away from a camera, there is a possibility that auser does not notice a region of interest on a 3D model or an inspectionimage. In order to surely notify a user of the position of the region ofinterest, the display control unit 182 may display the region ofinterest larger than its original size.

FIG. 14 shows an example of an inspection image displayed on the displayunit 5 in Step S204. An inspection image I₂₁ is displayed on the displayunit 5 and a plurality of regions R21 of interest are displayed on theinspection image I₂₁. The region R21 of interest includes a cooling holeCH11. The region R21 of interest is set larger than the cooling holeCH11. In order to make FIG. 14 easier to understand, the regions R21 ofinterest corresponding to only some of a plurality of cooling holes CH11are shown.

After Step S204, the camera position-posture estimation unit 184estimates the position and the posture of a camera when the cameraacquires an inspection image on the basis of at least one inspectionimage and the 3D model (Step S205). The processing for estimating theposition and the posture of a camera is generally called aperspective-n-point problem (PnP problem). By solving the PnP problemusing a known technique (for example, open source software or the like),it is possible to estimate the position and the posture of a camera.

The present invention is not limited to a method of estimating theposition and the posture of a camera by solving the PnP problem. Forexample, the camera position-posture estimation unit 184 may estimatethe position and the posture of a camera by using a sensor incorporatedin a scope unit of the endoscope. The sensor is an acceleration sensor,a gyro sensor, a magnetic sensor, or the like. In addition, the cameraposition-posture estimation unit 184 may estimate the position and theposture of a camera by combining a variety of sensors to improve theaccuracy of solving the PnP problem and shorten processing time.Therefore, the camera position-posture estimation unit 184 may estimatethe position and the posture of a camera on the basis of at least theinspection image and the 3D model.

In the example described below, it is assumed that the cameraposition-posture estimation unit 184 estimates the position and theposture of a camera by solving the PnP problem without using a sensor.The camera position-posture estimation unit 184 estimates the positionand the posture of a camera on the basis of only the inspection image.

After Step S205, the position identification unit 185 identifies aposition on the 3D model corresponding to a pixel of the inspectionimage (Step S206). The pixel subjected to be processed in Step S206 is apixel for which each piece of processing after Step S206 has not beenexecuted yet.

Details of Step S206 will be described with reference to FIG. 15. FIG.15 shows a method of identifying a position on the 3D model. A 3D modelMO11 has a tube shape. A cross section passing through the center axisof the 3D model MO11 is shown in FIG. 15. A state in which the insertionunit 2 is virtually inserted into the 3D model MO11 is shown in FIG. 15.

In Step S206, a position Pc of a camera with respect to the 3D modelMO11 is estimated. A region R31 of interest is set on the 3D model MO11.For example, the region R31 of interest corresponds to the region R1 ofinterest in FIG. 6. A plane PL11 is a virtual image plane on aperspective projection model. The plane PL11 is calculated on the basisof the position Pc and the posture of the camera estimated in Step S206.The position of the plane PL11 corresponds to a pixel of an inspectionimage. Hereinafter, the position of the plane PL11 is called a pixel.

A straight line that passes through the position Pc and each pixel canbe defined. Hereinafter, this straight line is called a visual line. Avisual line L11 passes through the position Pc and a pixel PX11. Avisual line L12 passes through the position Pc and a pixel PX12. Avisual line L13 passes through the position Pc and a pixel PX13. Aftereach visual line is calculated, it is possible to calculate anintersection point of each visual line and the 3D model MO11. A positionPO11 is an intersection point of the visual line L11 and the 3D modelMO11. A position PO12 is an intersection point of the visual line L12and the 3D model MO11. A position PO13 is an intersection point of thevisual line L13 and the 3D model MO11. Each of the position PO11, theposition PO12, and the position PO13 is a point on the inner wall of the3D model MO11. It is possible for the position identification unit 185to identify a position on the 3D model MO11 corresponding to coordinates(subject information) on an inspection image by calculating the positionPO11 and the like.

After Step S206, the determination unit 186 determines whether or notpredetermined information has been set at the position identified inStep S206. In this way, the determination unit 186 determines whether ornot the position identified in Step S206 has been observed (Step S207).The predetermined information represents that a position on the 3D modelhas been observed.

In Step S211 described later, it is determined whether or not a positionon the 3D model has been observed. When it is determined that a positionon the 3D model has been observed, information that represents that theposition has been observed is set to the position in Step S212. Thedetermination unit 186 determines whether or not the information hasbeen set to a position on the 3D model in Step S207. When theinformation has been set to a position on the 3D model, thedetermination unit 186 determines that the position has been observed.When the information has not been set to a position on the 3D model, thedetermination unit 186 determines that the position has not beenobserved.

When the determination unit 186 determines that the position identifiedin Step S206 has been observed, Step S213 is executed. When thedetermination unit 186 determines that the position identified in StepS206 has not been observed, the determination unit 186 reads an indexand a threshold value at the position from the RAM 14 (Step S208). Forexample, when the position PO13 is identified in Step S206, thedetermination unit 186 reads an index and a threshold value that hasbeen set to the position PO13 from the RAM 14. The position PO13 isincluded in the region R31 of interest corresponding to the region R1 ofinterest in FIG. 6. For this reason, the determination unit 186 reads anindex and a threshold value that has been set to the region R1 ofinterest from the RAM 14. The position PO11 and the position PO12 areincluded in a general region. When the position PO11 or the positionPO12 is identified in Step S206, the determination unit 186 reads anindex and a threshold value that has been set to the general region inFIG. 6 from the RAM 14.

After Step S208, the determination unit 186 calculates an index value atthe position identified in Step S206 (Step S209). Specifically, thedetermination unit 186 calculates an index value by using the inspectionimage, the position of the camera, the posture of the camera, and the 3Dmodel. The determination unit 186 calculates a specific value of theindex read in Step S208.

For example, the determination unit 186 calculates a value of the indexA (object distance) shown in FIG. 6 on the basis of the position of thecamera, the posture of the camera, and the 3D model. For example, thedetermination unit 186 calculates a value of the object distance at theposition PO11 by calculating the distance between the position Pc andthe position PO11 on the 3D model MO11 shown in FIG. 15. Thedetermination unit 186 calculates a value of each of the index B(stoppage time), the index C (image luminance), and the index D(position within a visual field) on the basis of the inspection image.

The position PO13 shown in FIG. 15 is included in the region R1 ofinterest shown in FIG. 6. When the position PO13 is identified in StepS206, the determination unit 186 calculates a value of each of the fourindices shown in FIG. 6. The position PO11 and the position PO12 shownin FIG. 15 are included in a general region. When the position PO11 orthe position PO12 is identified in Step S206, the determination unit 186calculates only a value of each of the index A and the index C shown inFIG. 6.

After Step S209, the determination unit 186 compares the index valuecalculated in Step S209 with the threshold value read in Step S208 (StepS210).

After Step S210, the determination unit 186 determines whether or notthe index value is greater than the threshold value. In this way, thedetermination unit 186 determines whether or not the position identifiedin Step S206 has been observed (Step S211). When a plurality of indexvalues are calculated in Step S209, the determination unit 186 compareseach index value with the threshold value in Step S210. Further, thedetermination unit 186 determines in Step S211 as follows. When all ofthe plurality of index values are greater than the threshold value, thedetermination unit 186 determines that the index value is greater thanthe threshold value. When at least one of the plurality of index valuesis less than or equal to the threshold value, the determination unit 186determines that the index value is not greater than the threshold value.

When only one of the plurality of index values is greater than thethreshold value, the determination unit 186 may determine that the indexvalue is greater than the threshold value. A method of determination inStep S211 is not limited to the above-described method. Thedetermination unit 186 may perform the above-described determination onthe basis of the logical product or the logical sum of a determinationresult obtained by comparing each index value with the threshold value,or on the basis of a combination of the logical product and the logicalsum.

When the determination unit 186 determines that the index value isgreater than the threshold value, the determination unit 186 generatesobservation information that represents that the position identified inStep S206 has been observed. In addition, the determination unit 186sets information that represents that the position identified in StepS206 has been observed to the position (Step S212).

After Step S212, the display control unit 182 generates a graphic imagesignal of the observation information. Thereafter, the video signalprocessing circuit 12 combines the graphic image signal and a videosignal and outputs the combined video signal to the display unit 5. Thedisplay unit 5 displays the observation information on the 3D model. Inthis way, the display unit 5 notifies a user that the positionidentified in Step S206 has been observed (Step S213).

When the determination unit 186 determines that the index value is notgreater than the threshold value, the determination unit 186 generatesobservation information that represents that the position identified inStep S206 has not been observed (Step S214).

After Step S214, the display control unit 182 generates a graphic imagesignal of the observation information. Thereafter, processing similar tothe processing in Step S213 is executed and the display unit 5 displaysthe observation information on the 3D model. In this way, the displayunit 5 notifies a user that the position identified in Step S206 has notbeen observed (Step S215).

FIG. 16 shows an example of an image displayed on the display unit 5 inStep S215. As with the image shown in FIG. 13, a 3D model M11 isdisplayed on the display unit 5 and a region R11 of interest and aregion R12 of interest are displayed on the 3D model M11. Further, aregion R13 that has been observed is displayed on the 3D model M11. Theregion R13 corresponds to the observation information. For example, theregion R13 is displayed in a color different from that of each of theregion R11 of interest and the region R12 of interest. A region otherthan the region R13 in the 3D model M11 has not been observed. Theregion R13 may be translucently displayed such that a user can visuallyrecognize the 3D model M11. As long as a user can distinguish between aregion that has been observed and a region that has not been observed,any display method may be used.

The observation information may be superimposed on the inspection image.FIG. 17 shows an example of an inspection image displayed on the displayunit 5 in Step S215. An inspection image I31 is displayed on the displayunit 5 and a region R31 of interest is displayed on the inspection imageI31. Further, a region R32 and a region R33 that have been observed aredisplayed on the inspection image I31. The region R32 and the region R33correspond to the observation information. A region other than theregion R32 and the region R33 in the inspection image I31 has not beenobserved.

After Step S213 or Step S215, the main control unit 180 determineswhether or not the processing of Steps S206 to S210 has been executedfor all the pixels subjected to be processed (Step S216). When the maincontrol unit 180 determines that the processing has not been executedfor some of the pixels, Step S206 is executed.

When it is determined that the position on the 3D image has beenobserved in Step S207, Step S213 is executed without executing theprocessing for determining whether or not the position has beenobserved. After it is determined that the position on the 3D image hasbeen observed, there is no case in which it is determined that theposition has not been observed. When a new inspection image isprocessed, the setting that the position has been observed continues.Step S207 may be executed at any timing before Step S214 is executed.

In the above-described example, an index value is calculated for eachpixel and observation information is generated for each pixel. An indexvalue may be calculated for a plurality of pixels and observationinformation may be generated for the plurality of pixels. Pixels forwhich the above-described processing is executed may be thinned out. Theabove-described processing may be omitted for a pixel at the edge of thevisual field.

When the main control unit 180 determines that the processing has beenexecuted for all the pixels, the main control unit 180 determineswhether or not the operation unit 4 has accepted completion ofinspection from a user (Step S217). When the main control unit 180determines that the operation unit 4 has not accepted the completion ofinspection from a user, Step S203 is executed. When the main controlunit 180 determines that the operation unit 4 has accepted thecompletion of inspection from a user, the inspection is completed.

The order of processing executed by the CPU 18 a is not limited to theorder shown in FIG. 11 and FIG. 12. For example, after the position andthe posture of the camera are estimated in Step S205, the inspectionimage may be displayed in Step S204.

A method of operating an observation device according to each aspect ofthe present invention includes an image display step, an estimationstep, an identification step, a determination step, and an observationinformation display step. The display control unit 182 displays an imageof a subject inside the observation object on the display unit 5 in theimage display step (Step S204). The camera position-posture estimationunit 184 estimates the position of the imaging device 28 and the postureof the imaging device 28 in the estimation step (Step S205). Theposition and the posture are a position and a posture, respectively,when the imaging device 28 acquires the image. In the identificationstep (Step S206), the position identification unit 185 identifies theposition on the 3D model corresponding to a pixel of the image on thebasis of the position and the posture that are estimated in theestimation step. The determination unit 186 determines whether or notthe imaging condition that has been set in the region of interest (firstregion) or the general region (second region) including the position onthe 3D model is satisfied in the determination step (Step S211). In theobservation information display step (Step S213 and Step S215), thedisplay control unit 182 displays observation information on the displayunit 5 on the basis of a result of determination executed by thedetermination unit 186. The observation information represents whetheror not the region of interest or the general region including theposition on the 3D model has been observed.

When the determination unit 186 determines that the imaging condition issatisfied, the display control unit 182 displays observation informationthat represents that the imaging condition is satisfied on the displayunit 5 in the observation information display step (Step S213). On theother hand, when the determination unit 186 determines that the imagingcondition is not satisfied, the display control unit 182 may displayobservation information that represents that the region of interest orthe general region including the position on the 3D model has not beenobserved on the display unit 5 in the observation information displaystep (Step S215).

In the first embodiment, when the imaging condition suitable for theregion of interest is satisfied, it is determined that the region ofinterest has been observed. The endoscope device 1 can determine whetheror not a region of a subject has been observed on the basis of thecondition of each region. Since a result of the determination isvisualized, the endoscope device 1 can offer an observation assistancefunction having the high reliability to a user.

(First Modified Example of First Embodiment)

A first modified example of the first embodiment of the presentinvention will be described. The PC 41 executes the processing shown inFIG. 5 in the first embodiment. The CPU 18 a of the endoscope device 1includes at least part of the functions of the CPU 43 of the PC 41 inthe first modified example of the first embodiment. The CPU 18 a of theendoscope device 1 executes at least part of the processing shown inFIG. 5 and FIG. 10.

For example, the CPU 18 a of the endoscope device 1 may include acondition setting unit (imaging condition setting unit) similar to thecondition setting unit 437 of the CPU 43. The condition setting unit ofthe CPU 18 a sets an imaging condition to each of a region of interestand a general region in an imaging condition setting step (Step S107).

The CPU 18 a of the endoscope device 1 may include a region acceptanceunit similar to the region acceptance unit 434 of the CPU 43. The regionacceptance unit of the CPU 18 a sets a region of interest and a generalregion to a 3D model in a region setting step (Step S105). The regionsetting step is executed before the imaging condition setting step isexecuted.

The CPU 18 a of the endoscope device 1 may include a 3D model generationunit (processing unit) similar to the 3D model generation unit 432 ofthe CPU 43. The 3D model generation unit of the CPU 18 a detects afeature point in each of a plurality of images acquired by the imagingdevice 28 in a feature point detection step (Step S103 a). The 3D modelgeneration unit of the CPU 18 a associates the feature point in each ofthe plurality of images between the images in an association step (StepS103 b). The 3D model generation unit of the CPU 18 a calculates theposition of the imaging device 28 and the posture of the imaging device28 on the basis of the feature point in a calculation step (Step S103c). The 3D model generation unit of the CPU 18 a generates a 3D model onthe basis of the position of the imaging device 28 and the posture ofthe imaging device 28 in a generation step (Step S103 d). Before theregion setting step is executed, the feature point detection step, theassociation step, the calculation step, and the generation step areexecuted.

(Second Modified Example of First Embodiment)

A second modified example of the first embodiment of the presentinvention will be described. In the first embodiment, the case in whicha user designates a region of interest is described. The presentinvention is not limited to the case in which a region of interestdesignated by a user is set. The PC 41 or the endoscope device 1 mayautomatically detect a region of interest in an image and may set thedetected region of interest to a 3D model.

In the first embodiment, the case in which a user designates an imagingcondition is described. The present invention is not limited to the casein which an imaging condition designated by a user is set. The PC 41 orthe endoscope device 1 may automatically generate an imaging conditionand may set the generated imaging condition to a 3D model.

The PC 41 shown in FIG. 3 is changed to a PC 41 a shown in FIG. 18. FIG.18 shows a functional configuration of the PC 41 a. The sameconfiguration as the configuration shown in FIG. 3 will not bedescribed.

The CPU 43 shown in FIG. 3 is changed to a CPU 43 a. The functions ofthe CPU 43 a are constituted by a video selection unit 430, a conditionacceptance unit 431, a 3D model generation unit 432, a display controlunit 433, a region detection unit 438, an index generation unit 439, athreshold value generation unit 440, and a condition setting unit 437.At least one of the blocks in the CPU 43 a shown in FIG. 18 may beconstituted by a circuit different from the CPU 43 a.

Each unit in the CPU 43 a may be constituted by at least one of aprocessor and a logic circuit. Each unit in the CPU 43 a may include oneor a plurality of processors. Each unit in the CPU 43 a may include oneor a plurality of logic circuits.

The region detection unit 438 automatically detects a region of interestand a general region on the basis of an image acquired by the imagingdevice 28. For example, the region detection unit 438 detects a regionof interest by using a technical means such as machine learning. Theregion detection unit 438 classifies the detected region as a region ofinterest and classifies a region other than the region of interest as ageneral region. The region detection unit 438 sets the region ofinterest and the general region to a 3D model generated by the 3D modelgeneration unit 432. Each region set by the region detection unit 438 isstored on the memory 47.

In processing in which the region detection unit 438 detects a region ofinterest and a general region, input of information that represents aregion of interest and a general region from a user is not accepted.Therefore, even if a user operates the operation unit 4, the user isunable to input information that designates each region to the operationunit 4. The operation unit 4 stops acceptance of the information.

The index generation unit 439 automatically generates an index on thebasis of an image acquired by the imaging device 28. Specifically, theindex generation unit 439 identifies a type of a subject seen in aregion of interest in an inspection image. For example, the type of thesubject is a cooling hole, an edge, a welding trace of a pipe, or thelike. The index generation unit 439 generates an index suitable for theidentified type of the subject. The threshold value generation unit 440automatically generates a threshold value of the index generated by theindex generation unit 439.

In processing in which the index generation unit 439 generates an indexand in processing in which the threshold value generation unit 440generates a threshold value, input of an imaging condition from a useris not accepted. Therefore, even if a user operates the operation unit4, the user is unable to input information that designates an index anda threshold value to the operation unit 4. The operation unit 4 stopsacceptance of the imaging condition.

FIG. 19 shows a procedure of processing executed by the PC 41 a. Thesame processing as the processing shown in FIG. 5 will not be described.Hereinafter, an example in which an inspection image included in aninspection moving image selected in Step S101 is used will be described.When the processing described below is executed, any one of a movingimage and a still image may be used.

After Step S104, the region detection unit 438 detects a region ofinterest. The region detection unit 438 classifies a region other thanthe region of interest as a general region. The region detection unit438 sets the region of interest and the general region to the generated3D model (Step S111).

For example, the region detection unit 438 detects a first region froman inspection image. The detected first region is similar to a region ofall or part of an image used for measurement by a user in the past.Alternatively, the region detection unit 438 detects a second regionfrom the inspection image. Annotation (meta data) embedded in an imageor a specific position in an image as an inspection record by a user ispresent in the detected second region. Alternatively, the regiondetection unit 438 detects a third region from another inspection imageprepared for region setting. The detected third region is similar to aregion designated on the inspection image by a user.

A specific example of a procedure for setting a region of interest byusing the machine learning will be described. A region that should bedetected as a region of interest is defined. The region is a region forwhich measurement has actually been performed in a measurement image ora region to which annotation has been given. A user manages themeasurement image on the basis of the past inspection. The measurementimage is input to the PC 41 a as teacher data with a correct answer thatshould be detected as a region of interest of the machine learning. Theregion detection unit 438 learns a region of interest by using theteacher data. When a certain inspection image is input to the PC 41 a,the region detection unit 438 performs detection of a region ofinterest. If there is a region of interest in a subject seen in theinspection image, it is possible for the region detection unit 438 tooutput the region of interest.

The region detection unit 438 may execute more complicated processingthan simple processing for determining whether or not there is a regionof interest. For example, the region detection unit 438 may classify thetype of a detected region of interest. It is possible for the regiondetection unit 438 to classify a detected region of interest as a flawpresent at an edge portion of a blade, a flaw present in a cooling hole,or the like. By using this method, it is possible for the regiondetection unit 438 to automatically set a region of interest regardlessof an instruction from a user.

After Step S111, the index generation unit 439 automatically generatesan index of each of the region of interest and the general region. Thethreshold value generation unit 440 generates a threshold value of theindex of each region (Step S112). After Step S112, Step S107 isexecuted.

For example, the index generation unit 439 classifies the type of asubject seen in the region of interest by performing image processing.The index generation unit 439 may use the type of the subject classifiedby the region detection unit 438. An index and a threshold value arestored on the memory 47 for each type of a subject in advance. The indexgeneration unit 439 reads an index corresponding to the type of thesubject from the memory 47. This index is to be set to the region ofinterest. The threshold value generation unit 440 reads a thresholdvalue corresponding to the type of the subject from the memory 47. Thisthreshold value is to be set to the region of interest.

As with an index and a threshold value of the region of interest, anindex and a threshold value of the general region are stored on thememory 47 in advance. The index generation unit 439 reads an index ofthe general region from the memory 47 and the threshold value generationunit 440 reads a threshold value of the general region from the memory47.

The CPU 18 a of the endoscope device 1 may include at least part of thefunctions of the CPU 43 a of the PC 41 a. The CPU 18 a of the endoscopedevice 1 may execute at least part of the processing shown in FIG. 19.

For example, the CPU 18 a of the endoscope device 1 may include a regiondetection unit similar to the region detection unit 438 of the CPU 43 a.The CPU 18 a of the endoscope device 1 may include an index generationunit similar to the index generation unit 439 of the CPU 43 a. The CPU18 a of the endoscope device 1 may include a threshold value generationunit similar to the threshold value generation unit 440 of the CPU 43 a.

In the second modified example of the first embodiment, a region ofinterest and an imaging condition are automatically set. For thisreason, the burden of a user is eased.

Second Embodiment

A second embodiment of the present invention will be described. Theendoscope device 1 according to the second embodiment has threefunctions described below in addition to the function in the firstembodiment. The three functions are useful when a user inspects aninspection object. The endoscope device 1 does not need to have all ofthe three functions. A case in which the endoscope device 1 has only oneor two of the three functions described below is included in the scopeof the present invention.

When it is determined that a region of interest or a general region hasnot been observed, the endoscope device 1 notifies a user of a cause ofthe determination that the region has not been observed (firstfunction). The first function may include a function of notifying a userof a method of changing an imaging condition such that it is determinedthat a region of interest or a general region has been observed inconsideration of the cause of the determination that the region has notbeen observed. When it is determined that a position on a 3D model hasnot been observed and the position is behind a visual field of theimaging device 28, the endoscope device 1 notifies a user that there isa region of interest or a general region that has not been observed(second function). The endoscope device 1 automatically records aninspection image in which a region of interest that has been observed isseen (third function).

The CPU 18 a shown in FIG. 4 is changed to a CPU 18 b shown in FIG. 20.FIG. 20 shows a functional configuration of the CPU 18 b. The sameconfiguration as the configuration shown in FIG. 4 will not bedescribed.

The functions of the CPU 18 b are constituted by a main control unit180, an image acquisition unit 181, a display control unit 182, a 3Dmodel selection unit 183, a camera position-posture estimation unit 184,a position identification unit 185, a determination unit 186, and arecording unit 187. At least one of the blocks shown in FIG. 20 may beconstituted by a circuit other than the CPU 18 b.

Each unit shown in FIG. 20 may be constituted by at least one of aprocessor and a logic circuit. Each unit shown in FIG. 20 may includeone or a plurality of processors. Each unit shown in FIG. 20 may includeone or a plurality of logic circuits.

An imaging condition includes a plurality of conditions. Therefore, animaging condition includes a plurality of indices and a threshold valueof each index. When the determination unit 186 determines that at leastone condition included in the plurality of conditions is not satisfied,the display control unit 182 displays condition information thatrepresents the at least one condition on the display unit 5. In thisway, the display unit 5 notifies a user of the at least one conditionthat is a cause of the determination that a position on a 3D model hasnot been observed.

The display control unit 182 may display information that represents aspecific method of changing an imaging condition such that it isdetermined that the position on the 3D model has been observed on thedisplay unit 5. For example, when the object distance is large and it isdetermined that the position on the 3D model has not been observed, thedisplay control unit 182 may display a message “The present objectdistance is OO mm. Please perform observation such that the objectdistance becomes XX mm.” on the display unit 5. When the type of anoptical adaptor is the cause of the determination that the position onthe 3D model has not been observed, the display control unit 182 maydisplay a message “Please change the type of the optical adaptor fromdirect view to side view.” on the display unit 5.

RAM 14 stores a non-observed position that is a position on a 3D modelfor which it is determined that an imaging condition is not satisfied.When the non-observed position is behind a visual field of the imagingdevice 28, the display control unit 182 displays first notificationinformation that represents that there is a region of interest or ageneral region including the non-observed position on the display unit5. The position behind the visual field of the imaging device 28 is notin the visual field. The position behind the visual field of the imagingdevice 28 is a position on a 3D model away from a position within thevisual field toward a viewpoint.

For example, a position on a 3D model corresponding to a firstinspection image is identified and it is determined that the positionhas not been observed. The position is stored on the RAM 14. After thefirst inspection image is acquired, a second inspection image isacquired. When there is a non-observed position behind a visual field ofthe imaging device 28 at the time of acquiring the second inspectionimage, the display control unit 182 displays the first notificationinformation on the display unit 5.

When the determination unit 186 determines that an imaging conditionthat has been set to a region of interest is satisfied, the recordingunit 187 records an inspection image in which a region corresponding tothe region of interest on a 3D model is seen. For example, the recordingunit 187 records the inspection image on the memory card 42. Therefore,the inspection image in which the region of interest that has beenobserved is seen is recorded.

The processing shown in FIG. 12 is changed to processing shown in FIG.21. A procedure of processing executed during inspection will bedescribed with reference to FIG. 21. FIG. 21 shows a procedure ofprocessing executed by the CPU 18 b. The same processing as theprocessing shown in FIG. 12 will not be described.

In a case in which the determination unit 186 determines that an indexvalue is not greater than a threshold value in Step S211, a non-observedposition exists. In such a case, it is notified to a user that aposition on a 3D model has not been observed in Step S215.

After Step S215, the determination unit 186 notifies the display controlunit 182 of an index of which a value does not exceed a threshold valuein Step S211. The display control unit 182 generates a graphic imagesignal for displaying the non-observed position in a color correspondingto the index. Thereafter, the video signal processing circuit 12combines the graphic image signal and a video signal and outputs thecombined video signal to the display unit 5. The display unit 5 displaysthe non-observed position on the 3D model in a predetermined color. Thenon-observed position is displayed in a color corresponding to apredetermined index. The color corresponds to the condition information.In this way, the display unit 5 notifies a user of a cause of thedetermination that the position on the 3D model has not been observed(Step S221). After Step S221, Step S216 is executed.

Details of Step S221 will be described with reference to FIG. 22. FIG.22 shows a method of notifying a user of a cause of the determinationthat the position on the 3D model has not been observed. A 3D model MO41has a tube shape. A cross section passing through the center axis of the3D model MO41 is shown in FIG. 22. A state in which the insertion unit 2is virtually inserted into the 3D model MO41 is shown in FIG. 22.

A region R41 is a region for which it is determined by the determinationunit 186 that the region has been observed. A region R42 is a region forwhich a value of the index A in FIG. 6 does not exceed a threshold valueand it is determined by the determination unit 186 that the region hasnot been observed. A region R43 is a region for which a value of theindex B in FIG. 6 does not exceed a threshold value and it is determinedby the determination unit 186 that the region has not been observed.

In the first embodiment, the display unit 5 displays observationinformation that represents that the region R42 and the region R43 havenot been observed. In the first embodiment, the region R42 and theregion R43 are displayed in a color different from that of the regionR41. For this reason, a user can know that the region R42 and the regionR43 have not been observed. In the first embodiment, the color of theregion R42 and the color of the region R43 are the same. For thisreason, a user is unable to know why it is determined that the regionR42 and the region R43 have not been observed. A user is unable todetermine under what kind of conditions an observation object should beobserved when the observation object is observed again. Therefore, auser is unable to know how the imaging condition is to be changed.

In the second embodiment, the region R42 is displayed in a colorassigned to the index A and the region R43 is displayed in a colorassigned to the index B. The color assigned to each of the index A andthe index B is different from the color for displaying a region forwhich it is determined that the region has been observed. A user canknow a cause of the determination that a region has not been observed.When the region is observed again, a user can observe a subject payingattention to the cause. In addition, a user can know how the imagingcondition is to be changed.

A character or the like that represents an index of which a value doesnot exceed a threshold value may be displayed on the display unit 5. Aslong as a user can understand a cause of the determination that a regionhas not been observed, any notification method may be used.

When the main control unit 180 determines that the processing has beenexecuted for all the pixels in Step S216, the determination unit 186determines whether or not the non-observed position is behind a visualfield of the imaging device 28 (Step S222). When the non-observedposition is not behind the visual field of the imaging device 28, StepS224 is executed.

When the non-observed position is behind the visual field of the imagingdevice 28, the display control unit 182 generates a graphic image signalfor displaying first notification information that represents that thereis a region including a non-observed position. Thereafter, the videosignal processing circuit 12 combines the graphic image signal and avideo signal and outputs the combined video signal to the display unit5. The display unit 5 displays the first notification information on the3D model. The first notification information is a character or the like.In this way, the display unit 5 notifies a user that there is a regionincluding a non-observed position behind the visual field of the imagingdevice 28 (Step S223).

The fact that there is a region including a non-observed position may benotified to a user by voice. As long as a user can understand that thereis a region including a non-observed position, any display method may beused.

In Step S211, it is determined whether or not a position on the 3D modelhas been observed. When it is determined that the position on the 3Dmodel has not been observed, information that represents that theposition has not been observed is set to the position in Step S214. Thedetermination unit 186 determines whether or not the position to whichthe information has been set exists on the 3D model in Step S222. Whenthe position exists, the determination unit 186 determines that theposition is a non-observed position. Further, the determination unit 186determines whether or not the non-observed position is behind a visualfield of the imaging device 28 in Step S222.

Details of Step S222 will be described with reference to FIG. 23. FIG.23 shows a method of detecting a non-observed position behind a visualfield of the imaging device 28. A 3D model MO51 has a tube shape. Across section passing through the center axis of the 3D model MO51 isshown in FIG. 23. A region R51 is a region for which it is determined bythe determination unit 186 that the region has been observed. A positionP51 is a position of a camera estimated on the basis of a firstinspection image. A position P52 is a position of the camera estimatedon the basis of a second inspection image acquired following the firstinspection image. The position P52 is the current position of thecamera. A motion vector V51 represents the motion between the positionP51 and the position P52.

When the position P51 and the position P52 are estimated, it is possibleto calculate the motion vector V51. The motion vector V51 represents atraveling direction of the camera. A line L51 and a line L52 representvisual lines passing through both ends of the visual field of theimaging device 28. When the first inspection image acquired at the timeat which the camera is at the position P51 is processed, it isdetermined that a region R52 has not been observed. When the secondinspection image acquired at the time at which the camera is at theposition P52 is processed, it is also determined that a region R52 hasnot been observed. When the camera is at the position P52, a region R52a is out of the visual field of the imaging device 28. The region R52 ais part of the region R52. The region R52 a has not been observed and isbehind the visual field of the imaging device 28.

In a case in which the camera continues to travel in the direction ofthe motion vector V51, there is a possibility that a user misses anopportunity of observing the inspection object. For this reason, it isnotified to a user that there is a region including a non-observedposition in Step S223.

After Step S223, the determination unit 186 determines whether or not aregion of interest that has been observed is seen in an inspectionimage. In addition, the determination unit 186 determines whether or notthe inspection image including the region of interest has been recorded(Step S224). When the determination unit 186 determines that the regionof interest that has been observed is not seen in the inspection image,Step S217 is executed. Alternatively, when the determination unit 186determines that the inspection image including the region of interestthat has been observed has been recorded, Step S217 is executed.

When the determination unit 186 determines that the region of interestthat has been observed is seen in the inspection image and thedetermination unit 186 determines that the inspection image includingthe region of interest has not been recorded, the recording unit 187records the inspection image (Step S225). In this way, the evidence thata user has observed the region of interest remains. After Step S225,Step S217 is executed.

Any one of a moving image and a still image may be recorded. In a casein which a moving image is recorded, a moving image from a timing atwhich a region of interest moves in to a timing at which the region ofinterest moves out may be recorded. In a case in which a still image isrecorded, a still image in which a region of interest is the largest ina visual field may be recorded. Alternatively, a still image having thelargest covering rate may be recorded. The covering rate is defined as aratio of a second area to a first area. The first area is the entirearea of the defined region of interest. The second area is the area of aregion of interest seen in an inspection image. There is no need torecord only one still image. A plurality of still images may berecorded.

The display control unit 182 may display a non-observed position on aninspection image in Step S221. The display control unit 182 may displayfirst notification information that represents that there is a regionincluding a non-observed position on an inspection image in Step S223.

When the determination unit 186 determines that an imaging condition setin a general region is satisfied, the recording unit 187 may record aninspection image in which a region corresponding to a general region ona 3D model is seen.

A method of operating an observation device according to each aspect ofthe present invention may include a condition information display step.An imaging condition includes a plurality of conditions. When thedetermination unit 186 determines that at least one condition is notsatisfied, the display control unit 182 displays condition informationthat represents the at least one condition on the display unit 5 in thecondition information display step (Step S221).

A method of operating an observation device according to each aspect ofthe present invention may include a first notification informationdisplay step. When a non-observed position is behind a visual field ofthe imaging device 28, the display control unit 182 displays firstnotification information that represents that there is a regionincluding the non-observed position on the display unit 5 in the firstnotification information display step (Step S223).

A method of operating an observation device according to each aspect ofthe present invention may include a recording step. When thedetermination unit 186 determines that an imaging condition set in aregion of interest is satisfied, the recording unit 187 records aninspection image in which a region corresponding to the region ofinterest is seen in the recording step (Step S225).

In the second embodiment, the endoscope device 1 offers following threefunctions. The endoscope device 1 notifies a user of a cause of thedetermination that a region has not been observed. The endoscope device1 notifies a user that a region that has not been observed is behind avisual field of the imaging device 28. The endoscope device 1 records aninspection image in which a region of interest that has been observed isseen. Since these functions are added, an inspection time can beexpected to be shortened. In addition, the endoscope device 1 can offeran observation assistance function having the higher reliability to auser.

Third Embodiment

A third embodiment of the present invention will be described. In thefirst embodiment, two states are discriminated. A first state is a statein which a region has been observed. A second state is a state in whicha region has not been observed. In the third embodiment, reliabilityaccording to the degree to which a region has been observed iscalculated. The reliability can be configured to represent any one oftwo or more states.

The endoscope device 1 according to the third embodiment includes theCPU 18 a shown in FIG. 4. The determination unit 186 calculatesreliability (evaluation value) that represents the degree to which animaging condition set in a region of interest or a general region issatisfied. The display control unit 182 displays the reliability asobservation information on the display unit 5. The observationinformation represents whether or not a region of interest or a generalregion including a position on a 3D model has been observed.

The determination unit 186 determines whether or not an imagingcondition is satisfied by comparing the reliability with a thresholdvalue. This threshold value is not necessarily the same as a thresholdvalue used for determination of an index value. When the determinationunit 186 determines that the imaging condition is not satisfied, thedisplay control unit 182 displays second notification information thatrepresents a region of interest or a general region including a positionon a 3D model has not been observed on the display unit 5.

The processing shown in FIG. 11 is changed to processing shown in FIG.24. The processing shown in FIG. 12 is changed to processing shown inFIG. 25. A procedure of processing executed during inspection will bedescribed with reference to FIG. 24 and FIG. 25. FIG. 24 and FIG. 25show a procedure of processing executed by the CPU 18 a. The sameprocessing as the processing shown in FIG. 11 and FIG. 12 will not bedescribed.

After Step S206, Step S208 is executed. Step S207 shown in FIG. 11 isnot executed.

After Step S209, the determination unit 186 calculates reliability onthe basis of the index value calculated in Step S209 and the thresholdvalue read in Step S208 (Step S231). The reliability is defined suchthat the reliability of a position on a 3D model becomes higher as thepossibility that a user has observed a region including the positionbecomes higher. In a case in which the reliability of a position on a 3Dmodel is high, the possibility that a user has observed a regionincluding the position is high. In a case in which the reliability of aposition on a 3D model is low, the possibility that a user has observeda region including the position is low. For example, the reliability hasa value ranging from 0% to 100%. In a case in which the reliability is100%, a region of interest or a general region has been observed.

For example, the determination unit 186 calculates reliability by usingfollowing Expression (4).

$\begin{matrix}{E_{t} = {\sum\limits_{i = 0}^{N}\;{w_{i}\frac{e_{i}}{{Th}_{i}}}}} & (4)\end{matrix}$

In Expression (4), reliability in a certain time point t is expressed asE_(t) and a weight attached to an i-th index is expressed as W_(i). InExpression (4), a value of the i-th index is expressed as e_(i) and athreshold value of the i-th index is expressed as Th_(i). Thedetermination unit 186 calculates reliability of each position on a 3Dmodel in the time point t. The determination unit 186 repetitivelycalculates the reliability of each position until inspection iscompleted.

In a case in which the maximum value of the reliability E_(t) is updatedfrom the start of inspection to the end of the inspection, the value isstored on the RAM 14. An example in which a point of an inspectionobject is paid attention to. It is assumed that the tip end 20 of theinsertion unit 2 moves from a first position to a second position andfurther moves to a third position. The first position and the thirdposition are far from a point of an inspection object. The secondposition is close to the point of the inspection object. In a case inwhich the tip end 20 of the insertion unit 2 moves like this, themaximum value of the reliability is held. Updating a value of thereliability to a low value is avoided. The determination unit 186 maycalculate the sum of reliability at each time point for each position ona 3D model. The sum may be defined as the final reliability.

After Step S231, the display control unit 182 generates a graphic imagesignal for displaying reliability. Thereafter, the video signalprocessing circuit 12 combines the graphic image signal and a videosignal and outputs the combined video signal to the display unit 5. Thedisplay unit 5 displays observation information on an inspection image.Each of a region of interest and a general region is displayed in acolor corresponding to the reliability of each region. The colorcorresponds to the observation information. In this way, the displayunit 5 notifies a user of the reliability of the position identified inStep S206 (Step S232). After Step S232, Step S216 is executed.

FIG. 26 shows an example of an inspection image displayed on the displayunit 5 in Step S232. An inspection image I41 is displayed on the displayunit 5. Further, each of a region of interest and a general region isdisplayed in a color corresponding to the reliability of each region.While reliability is displayed on an inspection image, there is apossibility that a pattern of a subject is hard to be seen. For thisreason, the endoscope device 1 may have a function of switching betweenon and off of display of reliability on the basis of an instruction froma user. There is no need for second notification information to besuperimposed on an inspection image. As long as a user can understandreliability, any display method may be used.

When the main control unit 180 determines that the operation unit 4 hasaccepted completion of inspection from a user in Step S217, thedetermination unit 186 determines whether or not a proper inspection hasbeen performed (Step S233). For example, the determination unit 186calculates a representative value of reliability of each positionincluded in a region of interest or a general region. A representativevalue is the average, the median, the maximum, or the like ofreliability of each position included in a region of interest or ageneral region. As long as a representative value is calculated on thebasis of reliability of each position included in a region of interestor a general region, the representative value may be any value. Thedetermination unit 186 compares the calculated value with a thresholdvalue. For example, a method of setting a threshold value of reliabilityis similar to the method of setting a threshold value of an index.

When the representative value is greater than the threshold value in allthe regions of interest, the determination unit 186 determines that aproper inspection has been performed. When there is a region of interestin which the representative value is not greater than the thresholdvalue, the determination unit 186 determines that a proper inspectionhas not been performed.

A threshold value of reliability may be prepared for each region. Forexample, a threshold value of reliability of a region of interest may be100% and a threshold value of reliability of a general region may be80%. A threshold value of reliability of a region of interest and athreshold value of reliability of a general region may be the same. Forexample, a threshold value of reliability of all the regions may be 80%.A method of determining whether or not a proper inspection has beenperformed is not limited to the above-described method. As long as it isdetermined whether or not a proper inspection has been performed, anymethod may be used.

When the determination unit 186 determines that a proper inspection hasbeen performed, the processing shown in FIG. 25 is completed. When thedetermination unit 186 determines that a proper inspection has not beenperformed, the determination unit 186 notifies the display control unit182 of information of a region that should be inspected again. Therepresentative value of reliability of the region is not greater thanthe threshold value. The display control unit 182 generates a graphicimage signal for displaying second notification information thatrepresents the region. Thereafter, the video signal processing circuit12 combines the graphic image signal and a video signal and outputs thecombined video signal to the display unit 5. The display unit 5 displaysthe second notification information on an inspection image. In this way,the display unit 5 notifies a user of the region that should beinspected again (Step S234). When Step S234 is executed, the processingshown in FIG. 25 is completed.

For example, a region that should be inspected again is displayed in acolor different from that of a region for which a proper inspection hasbeen performed. A region for which a proper inspection has not beenperformed may not be displayed and only a region that should beinspected again may be displayed. As long as a user can understand aregion that should be inspected again, any display method may be used.

The display control unit 182 may display reliability on a 3D model inStep S232. The display control unit 182 may display second notificationinformation on a 3D model in Step S234.

As described above, the determination unit 186 calculates reliabilitythat represents the degree to which an imaging condition that has beenset in a region of interest or a general region is satisfied in adetermination step (Step S231). The display control unit 182 displaysthe reliability as observation information on the display unit 5 in anobservation information display step (Step S232).

As described above, the determination unit 186 determines whether or notthe imaging condition is satisfied by comparing the reliability with athreshold value in a determination step (Step S233). A method ofoperating an observation device according to each aspect of the presentinvention may include a second notification information display step.When the determination unit 186 determines that the imaging condition isnot satisfied, the display control unit 182 displays second notificationinformation that represents a region of interest or a general regionincluding a position on a 3D model has not been observed on the displayunit 5 in the second notification information display step (Step S234).

In the third embodiment, reliability is calculated and displayed. Thereliability has any one of two or more values that represent the degreeto which a region has been observed. For this reason, the endoscopedevice 1 can correctly notify a user of a region that has beenthoroughly observed and a region that has not been thoroughly observed.

The endoscope device 1 automatically determines whether or not a properinspection has been performed. In a case in which the endoscope device 1determines that a proper inspection has been performed, a user can feelat ease and report completion of the inspection to a client, asupervisor, or the like. When the endoscope device 1 determines that aproper inspection has not been performed, the endoscope device 1notifies a user of a region that should be inspected again. For thisreason, a user can promptly start re-inspection.

While preferred embodiments of the invention have been described andshown above, it should be understood that these are examples of theinvention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. A method of operating an observation device, themethod comprising: an image display step; an estimation step; anidentification step; a determination step; and an observationinformation display step, wherein the observation device includes: aninsertion unit that includes an imaging device configured to acquire animage of a subject inside an observation object and is configured to beinserted into the observation object; a storage medium configured tostore an imaging condition that has been set in each of a first regionand a second region, the first region being part of a three-dimensionalmodel of the subject, the second region being other than the firstregion in the three-dimensional model, the imaging condition of thefirst region and the imaging condition of the second region beingdifferent from each other; and a processor, the processor displays theimage on a display in the image display step, the processor estimates,in the estimation step, a position of the imaging device and a postureof the imaging device, the position and the posture being a position anda posture, respectively, when the imaging device acquires the image, theprocessor identifies, in the identification step, a position on thethree-dimensional model corresponding to a pixel of the image on thebasis of the position and the posture that are estimated in theestimation step, the processor determines, in the determination step,whether or not the imaging condition that has been set in the firstregion or the second region including the position on thethree-dimensional model is satisfied, and the processor displays, in theobservation information display step, observation information on thedisplay on the basis of a result of the determination step, theobservation information representing whether or not the first region orthe second region including the position on the three-dimensional modelhas been observed.
 2. The method of operating the observation deviceaccording to claim 1, wherein the method further comprises an imagingcondition setting step in which the processor sets the imaging conditionto each of the first region and the second region.
 3. The method ofoperating the observation device according to claim 1, wherein themethod further comprises a region setting step in which the processorsets the first region and the second region to the three-dimensionalmodel.
 4. The method of operating the observation device according toclaim 3, wherein the method further comprises: a feature point detectionstep in which the processor detects a feature point in each of aplurality of images acquired by the imaging device; an association stepin which the processor associates the feature point between the imagesincluded in the plurality of images; a calculation step in which theprocessor calculates a position of the imaging device and a posture ofthe imaging device on the basis of the feature point; and a generationstep in which the processor generates the three-dimensional model on thebasis of the position and the posture calculated in the calculationstep, and before the region setting step is executed, the feature pointdetection step, the association step, the calculation step, and thegeneration step are executed.
 5. The method of operating the observationdevice according to claim 2, wherein input of the imaging condition froma user is not accepted in the imaging condition setting step.
 6. Themethod of operating the observation device according to claim 3, whereininput of information that represents the first region and the secondregion from a user is not accepted in the region setting step.
 7. Themethod of operating the observation device according to claim 1, whereinthe processor estimates the position and the posture on the basis of atleast the image and the three-dimensional model in the estimation step.8. The method of operating the observation device according to claim 1,wherein the processor estimates the position and the posture on thebasis of only the image in the estimation step.
 9. The method ofoperating the observation device according to claim 1, wherein thestorage medium is configured to store a non-observed position that isthe position on the three-dimensional model for which it is determinedthat the imaging condition is not satisfied, and the method furthercomprises a first notification information display step in which, whenthe non-observed position is behind a visual field of the imagingdevice, the processor displays first notification information thatrepresents that there is the first region or the second region includingthe non-observed position on the display.
 10. The method of operatingthe observation device according to claim 1, wherein the imagingcondition includes a plurality of conditions, and the method furthercomprises a condition information display step in which, when theprocessor determines that at least one condition included in theplurality of conditions is not satisfied, the processor displayscondition information that represents the at least one condition on thedisplay.
 11. The method of operating the observation device according toclaim 1, wherein the method further comprises a recording step in which,when the processor determines that the imaging condition that has beenset to the first region is satisfied, the processor records the image inwhich a region corresponding to the first region is seen.
 12. The methodof operating the observation device according to claim 1, wherein, whenthe processor determines that the imaging condition is satisfied, theprocessor displays, in the observation information display step, theobservation information representing that the first region or the secondregion including the position on the three-dimensional model has beenobserved on the display.
 13. The method of operating the observationdevice according to claim 1, wherein, when the processor determines thatthe imaging condition is not satisfied, the processor displays, in theobservation information display step, the observation informationrepresenting that the first region or the second region including theposition on the three-dimensional model has not been observed on thedisplay.
 14. The method of operating the observation device according toclaim 1, wherein the processor calculates, in the determination step, anevaluation value that represents a degree to which the imaging conditionthat has been set in the first region or the second region is satisfied,and the processor displays the evaluation value as the observationinformation on the display in the observation information display step.15. The method of operating the observation device according to claim14, wherein the processor determines whether or not the imagingcondition is satisfied by comparing the evaluation value with athreshold value in the determination step, and the method furthercomprises a second notification information display step in which, whenthe processor determines that the imaging condition is not satisfied,the processor displays second notification information that representsthe first region or the second region including the position on thethree-dimensional model has not been observed on the display.
 16. Anobservation device comprising: an insertion unit that includes animaging device configured to acquire an image of a subject inside anobservation object and is configured to be inserted into the observationobject; a storage medium configured to store an imaging condition thathas been set in each of a first region and a second region, the firstregion being part of a three-dimensional model of the subject, thesecond region being other than the first region in the three-dimensionalmodel, the imaging condition of the first region and the imagingcondition of the second region being different from each other; and aprocessor, wherein the processor is configured to display the image on adisplay, the processor is configured to estimate a position of theimaging device and a posture of the imaging device, the position and theposture being a position and a posture, respectively, when the imagingdevice acquires the image, the processor is configured to identify aposition on the three-dimensional model corresponding to a pixel of theimage on the basis of the estimated position and posture, the processoris configured to determine whether or not the imaging condition that hasbeen set in the first region or the second region including the positionon the three-dimensional model is satisfied, and the processor isconfigured to display observation information on the display on thebasis of a result of determination of the imaging condition, theobservation information representing whether or not the first region orthe second region including the position on the three-dimensional modelhas been observed.
 17. A non-transitory computer-readable recordingmedium saving a program for causing a computer of an observation deviceto execute: an image display step; an estimation step; an identificationstep; a determination step; and an observation information display step,wherein the observation device includes: an insertion unit that includesan imaging device configured to acquire an image of a subject inside anobservation object and is configured to be inserted into the observationobject; and a storage medium configured to store an imaging conditionthat has been set in each of a first region and a second region, thefirst region being part of a three-dimensional model of the subject, thesecond region being other than the first region in the three-dimensionalmodel, the imaging condition of the first region and the imagingcondition of the second region being different from each other, thecomputer displays the image on a display in the image display step, thecomputer estimates, in the estimation step, a position of the imagingdevice and a posture of the imaging device, the position and the posturebeing a position and a posture, respectively, when the imaging deviceacquires the image, the computer identifies, in the identification step,a position on the three-dimensional model corresponding to a pixel ofthe image on the basis of the position and the posture that areestimated in the estimation step, the computer determines, in thedetermination step, whether or not the imaging condition that has beenset in the first region or the second region including the position onthe three-dimensional model is satisfied, and the computer displays, inthe observation information display step, observation information on thedisplay on the basis of a result of the determination step, theobservation information representing whether or not the first region orthe second region including the position on the three-dimensional modelhas been observed.
 18. The non-transitory computer-readable recordingmedium according to claim 17, wherein the program causes the computer tofurther execute a region setting step in which the computer sets thefirst region and the second region to the three-dimensional model. 19.The non-transitory computer-readable recording medium according to claim17, wherein the program causes the computer to further execute: afeature point detection step in which the computer detects a featurepoint in each of a plurality of images acquired by the imaging device;an association step in which the computer associates the feature pointbetween the images included in the plurality of images; a calculationstep in which the computer calculates a position of the imaging deviceand a posture of the imaging device on the basis of the feature point;and a generation step in which the computer generates thethree-dimensional model on the basis of the position and the posturecalculated in the calculation step, and before the region setting stepis executed, the feature point detection step, the association step, thecalculation step, and the generation step are executed.
 20. Thenon-transitory computer-readable recording medium according to claim 17,wherein the computer estimates the position and the posture on the basisof at least the image and the three-dimensional model in the estimationstep.